Skip to content

Conversation

@delphine-demeulenaere
Copy link
Contributor

@delphine-demeulenaere delphine-demeulenaere commented Nov 6, 2025

Description

While there is currently the ability to run single tests and all tests there is no functionality to run a batch of related tests on a template.

I added an extra option to the run-test command to run all tests which contain a common string.

Overall, I understand the changes made and the steps taken to implement this fix. I should note, however, that most of the actual code modifications were applied via cursor. That said, testing indicates that these changes achieve the desired results.

Testing Instructions

silverfin run-test -b "string pattern" -h <template name>
f.e.: silverfin run-test -b "unit_3" -h be_pit_legal_structures

Author Checklist

  • Skip bumping the CLI version

Reviewer Checklist

  • PR has a clear title and description
  • Manually tested the changes that the PR introduces
  • Changes introduced are covered by tests of acceptable quality

@michieldegezelle
Copy link
Contributor

@coderabbitai full review

@coderabbitai
Copy link

coderabbitai bot commented Dec 8, 2025

✅ Actions performed

Full review triggered.

@coderabbitai
Copy link

coderabbitai bot commented Dec 8, 2025

Walkthrough

Adds a new --pattern option to filter tests in run-test and development-mode flows, enforces mutual exclusivity with --test, and threads the pattern through test-building and execution paths. Introduces per-test line-adjustment calculations when filtering and propagates adjustments to logging outputs.

Changes

Cohort / File(s) Summary
CLI pattern option
bin/cli.js
Adds --pattern option to commands (run-test, development-mode, status flows). Validates mutual exclusivity with --test and passes pattern into downstream calls.
Dev mode pattern threading
lib/cli/devMode.js
Updates watchLiquidTest signature to accept pattern = "" and forwards pattern to both runTestsWithOutput invocations; updates JSDoc.
Test runner pattern filtering & metadata
lib/liquidTestRunner.js
Adds filterTestsByPattern and integrates into buildTestParams; buildTestParams now returns { testParams, metadata: { lineAdjustments } }. Adds per-test line-adjustment calculations, threads pattern through runTests, runTestsWithOutput, and runTestsStatusOnly, and applies adjustments in processTestRunResponse, listErrors, and logging for reconciled/results/rollforwards.
Changelog & version
CHANGELOG.md, package.json
Adds 1.49.0 changelog entry describing pattern/batch testing and bumps package version to 1.49.0.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

🚥 Pre-merge checks | ✅ 2 | ❌ 1
❌ Failed checks (1 warning)
Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 25.00% which is insufficient. The required threshold is 80.00%. You can run @coderabbitai generate docstrings to improve docstring coverage.
✅ Passed checks (2 passed)
Check name Status Explanation
Title check ✅ Passed The title 'Enable liquid batch testing' clearly and concisely summarizes the main change: adding batch testing capability via a pattern filter to the run-test command.
Description check ✅ Passed The description addresses the core change (batch testing via pattern filtering) and includes testing instructions, though it lacks explicit issue linking and doesn't fully match the template structure.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing touches
  • 📝 Generate docstrings

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)
bin/cli.js (1)

449-485: CLI UX mismatch: PR says run-test -b, code implements run-test -p (and dev-mode uses -b).

Given the PR objective/example, I’d align the short flag across commands (or update docs). If you want -b everywhere:

-  .option("-p, --pattern <pattern>", "Run all tests that match this pattern (optional)", "")
+  .option("-b, --pattern <pattern>", "Run all tests that match this pattern (optional)", "")

Also: consider making the action async and await the runner calls to avoid unhandled rejections / edge-case early exit:

-  .action((options) => {
+  .action(async (options) => {
@@
-      liquidTestRunner.runTestsStatusOnly(options.firm, templateType, templateName, options.test, options.pattern);
+      await liquidTestRunner.runTestsStatusOnly(options.firm, templateType, templateName, options.test, options.pattern);
@@
-      liquidTestRunner.runTestsWithOutput(options.firm, templateType, templateName, options.test, options.previewOnly, options.htmlInput, options.htmlPreview, options.pattern);
+      await liquidTestRunner.runTestsWithOutput(options.firm, templateType, templateName, options.test, options.previewOnly, options.htmlInput, options.htmlPreview, options.pattern);
     }
   });
lib/liquidTestRunner.js (1)

103-193: Block testName + pattern at the library boundary (not just CLI).

If a non-CLI caller passes both, you’ll set tests: filteredContent but compute test_line from the unfiltered testIndexes, which can point to the wrong line. Add a guard:

 function buildTestParams(firmId, templateType, handle, testName = "", renderMode, pattern = "") {
+  if (testName && pattern) {
+    consola.error("You cannot use both testName and pattern at the same time");
+    process.exit(1);
+  }
   let relativePath = `./reconciliation_texts/${handle}`;
🧹 Nitpick comments (3)
lib/cli/devMode.js (1)

34-47: Guard against overlapping runs + unhandled rejections in chokidar callbacks.

chokidar.watch(...).on("change", async () => { await ... }) can (a) trigger multiple times per save and (b) drop errors as unhandled promise rejections. Consider debouncing and wrapping the body in try/catch (or .catch(...)) and optionally serializing runs.

lib/liquidTestRunner.js (2)

34-101: findTestRows()-based indexing is too loose for reliable filtering/line-adjustment.

element.includes(testName) can match values/comments and skew indexes → wrong slicing and wrong lineAdjustments. Prefer matching actual YAML top-level keys (e.g., ^\s*<escapedName>\s*:).


461-555: Pattern propagation through runTests* is good; watch runTestsStatusOnly “undefined => PASSED” behavior.

Because runTests(...) returns undefined on various early-exit paths, runTestsStatusOnly currently treats that as PASSED. If that’s only intended for the “empty YAML” case, consider returning a sentinel/reason (or false) explicitly from runTests/buildTestParams so status-only can distinguish “no tests” from “couldn’t run”.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 836fc0f and b5802ec.

📒 Files selected for processing (3)
  • bin/cli.js (2 hunks)
  • lib/cli/devMode.js (2 hunks)
  • lib/liquidTestRunner.js (10 hunks)
🔇 Additional comments (4)
lib/cli/devMode.js (1)

9-22: watchLiquidTest(..., pattern = "") plumbing looks correct.

Signature + JSDoc update is consistent, and defaulting pattern to "" keeps backward behavior.

bin/cli.js (1)

675-710: development-mode --pattern is wired correctly; consider awaiting + enforce “pattern requires handle/account-template”.

Plumbing into devMode.watchLiquidTest(..., options.pattern) is consistent. Two follow-ups:

  • Make the action async and await devMode.watchLiquidTest(...) to surface errors deterministically.
  • Verify cliUtils.checkUniqueOption(["handle","updateTemplates","accountTemplate"], options) also enforces at least one of these; otherwise users could pass --pattern without --handle/--account-template despite the help text.
lib/liquidTestRunner.js (2)

220-270: Line adjustment hook in listErrors(...) looks consistent.

Passing lineAdjustment through and applying it only when line_number is numeric is clean.


308-382: Per-test line adjustment application is consistent; please validate API line-number semantics.

Using const lineAdjustment = lineAdjustments[testName] || 0; and applying it to reconciled/results/rollforwards is coherent if line_number is relative to the (possibly filtered) YAML file. Worth double-checking with a failing test in filtered mode to ensure numbers line up.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)
lib/liquidTestRunner.js (2)

20-32: findTestRows() key matching is unsafe (substring collisions) → can select wrong tests / wrong line numbers.

findTestRows() uses element.includes(testName) (Line 28). This will mis-detect YAML key lines when:

  • one test name is a substring of another (unit_1 matches unit_10:),
  • the test name appears in a value or comment before the key,
  • quoting/formatting varies.

That breaks filterTestsByPattern() segmentation (Line 44-79), test_line computation (Line 193), and lineAdjustments (Line 87-93).

Suggested fix: match YAML keys at line start (optionally quoted) instead of substring search.

 function findTestRows(testContent) {
   const options = { maxAliasCount: 10000 };
   const testYAML = yaml.parse(testContent, options);

   const testNames = Object.keys(testYAML);
   const testRows = testContent.split("\n");
   const indexes = {};
+
+  const escapeRegExp = (s) => s.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
   testNames.forEach((testName) => {
-    const index = testRows.findIndex((element) => element.includes(testName));
+    const keyRe = new RegExp(`^\\s*["']?${escapeRegExp(testName)}["']?\\s*:\\s*(#.*)?$`);
+    const index = testRows.findIndex((line) => keyRe.test(line));
     indexes[testName] = index;
   });
   return indexes;
 }

(If you need to support testName followed by inline YAML (key: { ... }), the regex still works.)

Also applies to: 34-101


103-196: Add testName XOR pattern guard to buildTestParams()

While the CLI prevents using both --test and --pattern together, the constraint should be enforced at the function level rather than relying solely on CLI validation. If buildTestParams() is called with both testName and pattern, finalTests gets filtered by pattern (lines 133–145) but testParams.test_line references the original testIndexes (line 193), causing a mismatch where the line number may target the wrong test.

Add this guard at the start of buildTestParams():

 function buildTestParams(firmId, templateType, handle, testName = "", renderMode, pattern = "") {
+  if (testName && pattern) {
+    consola.error("You cannot use both testName and pattern at the same time");
+    process.exit(1);
+  }
   let relativePath = `./reconciliation_texts/${handle}`;
📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between b5802ec and fdc8386.

⛔ Files ignored due to path filters (1)
  • package-lock.json is excluded by !**/package-lock.json
📒 Files selected for processing (3)
  • bin/cli.js (2 hunks)
  • lib/cli/devMode.js (2 hunks)
  • lib/liquidTestRunner.js (10 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • lib/cli/devMode.js
🔇 Additional comments (3)
lib/liquidTestRunner.js (2)

34-101: Line-adjustment plumbing looks consistent; please validate against API line_number semantics.

The lineAdjustments[testName] = originalIndex - filteredIndex approach (Line 87-93) and adding lineAdjustment when printing line_number (Line 229, 366, 374, 379) is coherent if the API returns line numbers relative to the submitted YAML file.

Request verification: run 1–2 failing expectations with and without --pattern and confirm the printed line points to the correct line in the original YAML.

Also applies to: 135-156, 223-273, 311-385


135-156: Nice UX improvement: matched tests are clearly listed.

bin/cli.js (1)

462-486: run-test pattern flag + mutual exclusivity check look good.

Copy link
Contributor

@michieldegezelle michieldegezelle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! 1 remaining comment (but approved already). Don't forget to update the CLI version number still

bin/cli.js Outdated
.option("-t, --test <test-name>", `Specify the name of the test to be run (optional). It has to be used together with "--handle"`, "")
.option("--html", `Get a html file of the template's input-view generated with the Liquid Test information (optional). It has to be used together with "--handle"`, false)
.option("--yes", "Skip the prompt confirmation (optional)")
.option("-b, --pattern <pattern>", `Run all tests that match this pattern (optional). It has to be used together with "--handle" or "--account-template"`, "")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you change the -b into -p?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adjusted!

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Fix all issues with AI agents
In @CHANGELOG.md:
- Line 5: Fix the typos in the changelog sentence: change "conatin" to "contain"
and add the missing space between "`silverfin run-test`" and "command" so the
sentence reads correctly (i.e., ensure it says "`silverfin run-test` command"
and uses "contain" instead of "conatin").
🧹 Nitpick comments (1)
bin/cli.js (1)

696-699: Consider extracting validation logic into a utility function.

The validation logic for preventing simultaneous use of --test and --pattern is duplicated between lines 470-473 (run-test) and lines 696-699 (development-mode). While this duplication is minor and follows existing inline validation patterns in this file, extracting it to cliUtils could improve maintainability if this constraint needs to change in the future.

♻️ Optional refactor to reduce duplication

In lib/cli/utils.js, add:

function checkTestPatternMutualExclusivity(options) {
  if (options.test && options.pattern) {
    consola.error("You cannot use both --test and --pattern options at the same time");
    process.exit(1);
  }
}

Then in bin/cli.js:

-    if (options.test && options.pattern) {
-      consola.error("You cannot use both --test and --pattern options at the same time");
-      process.exit(1);
-    }
+    cliUtils.checkTestPatternMutualExclusivity(options);

Apply the same change at both locations (lines 470-473 and 696-699).

📜 Review details

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between fdc8386 and 8b29510.

⛔ Files ignored due to path filters (1)
  • package-lock.json is excluded by !**/package-lock.json
📒 Files selected for processing (3)
  • CHANGELOG.md
  • bin/cli.js
  • package.json
✅ Files skipped from review due to trivial changes (1)
  • package.json
🔇 Additional comments (3)
bin/cli.js (3)

462-462: LGTM! Pattern option added consistently.

The --pattern option with -p short flag is now consistent across both run-test and development-mode commands, addressing previous feedback.


479-479: LGTM! Pattern properly threaded through execution paths.

The pattern parameter is correctly passed to all relevant function calls:

  • liquidTestRunner.runTestsStatusOnly (line 479)
  • liquidTestRunner.runTestsWithOutput (line 485)
  • devMode.watchLiquidTest (lines 706, 710)

This ensures the pattern filtering is applied consistently across all test execution paths.

Also applies to: 485-485, 706-706, 710-710


470-473: Empty string patterns are already handled correctly.

The code protects the filtering logic with an if (pattern) check at line 140 of lib/liquidTestRunner.js. Since empty strings are falsy in JavaScript, when pattern = "", the filtering is skipped and all tests run—the desired behavior. No changes needed.


All notable changes to this project will be documented in this file.
## [1.49.0] (07/01/2026)
In this version we are introducing Liquid batch/pattern testing. An extra option was added to the `silverfin run-test`command to run all tests which conatin a common string.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Fix typo in changelog entry.

Line 5 contains a typo: "conatin" should be "contain".

📝 Proposed fix
-In this version we are introducing Liquid batch/pattern testing. An extra option was added to the `silverfin run-test`command to run all tests which conatin a common string. 
+In this version we are introducing Liquid batch/pattern testing. An extra option was added to the `silverfin run-test` command to run all tests which contain a common string.

Note: Also added a missing space before "command".

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
In this version we are introducing Liquid batch/pattern testing. An extra option was added to the `silverfin run-test`command to run all tests which conatin a common string.
In this version we are introducing Liquid batch/pattern testing. An extra option was added to the `silverfin run-test` command to run all tests which contain a common string.
🤖 Prompt for AI Agents
In @CHANGELOG.md at line 5, Fix the typos in the changelog sentence: change
"conatin" to "contain" and add the missing space between "`silverfin run-test`"
and "command" so the sentence reads correctly (i.e., ensure it says "`silverfin
run-test` command" and uses "contain" instead of "conatin").

@delphine-demeulenaere delphine-demeulenaere merged commit 9b78203 into main Jan 7, 2026
3 checks passed
@delphine-demeulenaere delphine-demeulenaere deleted the liquid-batch-testing branch January 7, 2026 10:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants